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Theoretical  Concepts  for  Experimentation 


Developing  Theoretical  Concepts  for  Experimentation 

□  Parallel  approach  to  Experimentation  Thrust 

■  Overall  focus  on  augmenting  and  developing  macrocognition  concepts 

□  A)  Asking  how  can  we  enrich  understanding  of  foundational  collaboration 
concepts  through  empirical  studies? 

□  B)  Testing  macrocognition  concepts  through  refined  measurement 
techniques 


^SUMMIT 
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SIMULATION 
^TRAINING, 


□  A)  Presenting  overarching  concepts 
for  macrocognition  experiments 

■  Understanding  Problem  Space  and 
Impact  of  Task 

□  B)  Discussing  metrics  experimentation 
to  support  more  diagnostic  and 
construct  valid  understanding  of 
macrocognition 
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Overarching  Research  Questions  for  SUMMIT  Experimentation 

□  How  do  task  factors  alter  macrocog nitive  processes? 

■  How  do  changes  to  task  complexity  (e.g.,  low  versus  high  integrative  complexity) 
impact  macrocognitive  stages  and/or  processes? 

■  How  do  changes  to  task  structure  (e.g..  Hi-structured  versus  well-structured)  impact  the 
manner  in  which  teams  collaborate  as  they  work  through  macrocognitive  stages  and/or 
processes? 

□  How  does  distributed  interaction  influence  differing  elements  of  collaboration? 

■  What  happens  to  information  processing  within  and  across  teams  when  members  are 
not  all  co-iocated? 

■  How  do  changes  to  the  task  interact  with  distributed  interaction? 

□  What  is  the  impact  of  agent-based  team  members? 

■  How  does  inclusion  of  agents  supporting  certain  macrocognitive  processes  impact 
overall  stages  and/or  processes? 

■  Do  task  variations  interact  with  inclusion  of  agents  in  their  impact  on  macrocognition? 

□  Can  we  triangulate  on  macrocognitive  processes  through  improved 

measures? 

■  What  measures  provide  the  most  diagnostic  utility  as  to  assessing  macrocognition 
across  the  stages  of  collaborative  problem  solving? 
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Macrocognition  and  Experimentation  with  Task  Variation 

□  Background 

■  CKI  program  now  looking  at  macrocognition  in  varied  tasks 

□  SUMMIT  Goal 

■  Assess  how  variation  of  theoretically  important  factors,  within  a  given  testbed, 

alters  macrocognition 

□  Rationale 

■  Practical  Significance 

□  Research  across  variety  of  situational  factors  would  support 
understanding  and  improving  operational  performance 

■  Theoretical  Significance 

□  Research  on  macrocognition  would  benefit  from  further  integration  of 
cognitive  science  concepts 

□  Task  classifications  would  clarify 
influence  of  task  structure  and 
complexity  to  help  better 
understand  macrocognition 
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Developing  Theoretical  Concepts  for  Experimentation 

Macrocognition  —  Problem  Space  and  Influence  of  Task 

□  Understanding  problem  space  theory  in  context  of  macrocognition 
(Newell  &  Simon,  1972) 

■  the  mental  space  in  which  the  problem  solver  must  encode 
problem  elements  -  defining  goals,  rules  and  other  aspects  of 
the  situation...  [that]  represent: 

□the  initial  situation  presented 

□the  desired  goal  situation 

□various  intermediate  states, 
imagined  or  experienced 
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Macrocog nition  -  Problem  Space  and  Influence  of  Task 

Reifying  Problem  Space  Concept  through  Operationalization  of  Task  Variability 

■  Question  is  how  do  task  factors  alter  problem  space 

□  The  task  defines  the  “topology”  of  the  problem  space 

■  Dictates  paths  through  the  problem 
space  available  to  the  problem 
solver 

■  Some  successfully  lead  to  solution 

■  Collaborative  process  determines 
path  choice 

□  Experimentation  will  explore  how  this  alte 

■  Overarching  Hypothesis 

□  Differential  impact  of  task  manipulations  on  subcomponents  of 
macrocognition 

□  For  example,  there  will  changes  to  quantity  and  quality  of 
knowledge  building  when  task  is  more  ill-structured? 


macrocog nitive  processes 
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Macrocognition  -  Problem  Space  and  Influence  of  Task 


Theoretical  Issue  -  Problem  Space  and  Task  Complexity  (Wood,  1986) 

■  Component  Complexity 

□Amount  of  distinct  acts  associated  with  task  and  amount  of 
cues/problem  elements  to  be  processed 

■  Coordinative  Complexity 

□  Degree  to  which  task  variables 
need  to  be  integrated  for 
successful  task  completion 
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Macrocognition  -  Problem  Space  and  Influence  of  Task 

Theoretical  Issue  -  Problem  Space  and  Task  Structure  (Campbell,  1991) 

□  Determined  by  the  number  of  task  paths  to  follow  and/or  the  amount  of 
ambiguity  or  uncertainty  associated  with  the  paths. 


■  Multiple  Paths 

□  Degree  to  which  distinct  outcomes  are  possible  in  task  environment 


■  Degrees  of  Uncertainty 

□  Degree  to  which  task  alternatives 
are: 

■Ambiguous  as  to  the  path 
elements  and/or 

■  Differ  in  likelihood  of  occurring 
(i.e.,  amount  of  ambiguity 
associated  with  outcomes) 
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□Task  Complexity  -  Component  Complexity  '^  SUMMIT 

■  Amount  of  distinct  acts  associated  with  task  and  amount  of  cues/problem 
elements  to  be  processed 

■  MACRO-COG  missions  composed  of  several  operations 

□  Manipulating  number  of  operations  increases  component  complexity 

■  For  example,  plans  required  for  each  operation  -  the  more 
elements  required  in  the  plan  the  more  complexity 

□Also  number  of  resources, 
team  members,  and  rules 
will  be  varied  to 
manipulate  component 
complexity 


NEO-based  Scenario  Development 

Rebel  forces  attempting  to  overthrow  government. 

Generate  plans  to  aid  government  and  civilians. 

Iterative  Planning  Scenario 

Drapo  Islands 

,f)\  if 

S  Multiple  planning  events  at  different 
locations 

S  Various  constraints  including 
interdependencies  and  resource  limitations 

S  Five  team  members: 

i  h)  ;  - 

©  Weather/Environment 

Wm 

(2)  Supply  Specialist 

3  Transportation 

4  Intel  1  (Local) 

#  Intel  2  (Global) 

Theoretical  Concepts  for  Experimentation 

□Task  Complexity  -  Coordinative  Complexity  in  SUMMIT 

■  Degree  to  which  task  variables  need  to  be  integrated  for 
successful  task  completion 

■  MACRO-COG  allows  for  manipulating  interdependencies 
between  roles 

□For  example,  weather 
person  knows  critical  for 
equipment  person  (who 
needs  to  decide  if  it  is 
too  windy  to  use  a  UAV  for 
example) 

■  Scenario  creation  allows  for 
determining  amount  of  such 
interdependencies 


MACRO-COG  -  Establish  a  Land  Base 
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Task  Structure  -  Multiple  Paths  and  Degree  of  Uncertainty  i  n  SUMMIT 

■  MACRO-COG  allows  for  manipulations  of  resources 
□Quantity  and  variety  resources 

■  Alter  number  of  possible  plans 

■  Influence  number  of  possible  outcomes 
□Some  resources  are  information  resources 

■  Each  differing  degrees 
of  certainty  (e.g.,  going 
to  intel  and  weather 
roles) 

■  Accessing  information 
has  different  costs 
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Main  Role  Screen 


Summary  -  Concepts  for  Experimentation 


Experiment  Thrust  SUMMARY 

□  A)  Overarching  concepts  for  macrocognition  experiments 

■  How  do  task  factors  alter  macrocognitive  processes? 

■  How  does  distributed  interaction  influence  differing 
elements  of  collaboration? 

■  What  is  the  impact  of  agent-based  team  members? 

□  B)  Metrics  experimentation  to  support  more  diagnostic  and 
construct  valid  understanding  of  macrocognition 
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STRUCTURAL  MODEL  OF  TEAM  COLLABORATION 


Problem  Area 
Characteristics 


Collaborative  Situation 
Parameters: 


•  time  pressure 

•  information/knowledge 
uncertainty 

•  dynamic  information 

•  large  amount  of  knowledge 
(cognitive  overload) 

•  human-agent  interface 
complexity 


Team  Types 


•  asynchronous 

•  distributed 

•  culturally  diverse 

•  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 


Operational  Tasks 


•  team  decision  making,  COA 
selection 

•  develop  shared  understanding 

■  *  intelligence  analysis _ 


(team  data  processing) 


Collaboration  Stages 


Team 

*  ^Knowledge 


Base 
Constructio 


Team 

Consensus 


Outcome 
Evaluation 
and  Revision 


Collaboration 

Complete 

t  Yes 


Meta-Cognitive: 

•  individual  understanding  of 

problem  conditions 

•  individual  mental  model  develo^e^  p|an  t0  solve  problem 
of  situational  significance 

•  Information*  Processing : . 

•  problem  identification 

•  understanding  problem  task 


•  goal  development  j  •  track  team’s  mental  model  change?  compare  problem  solution 

■  team  mental  model  of  problem  •  understanding  remaining  items  to  a£aaist  §oa^s 

resolve 


•  establish  team  communication 
and  trust 

•  establish  data  filtering  methods 


goal  definition 

iterative  information  collectid 
&  analysis 

develop,  rationalize,  &  visualize 


team  negotiation  of  solution 
alternatives 
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•Knowledge  Building: 

•  problem  definition 

•  individual  task  knowledge 

•  individual  team  knowledge 


•  team  mental  model  of  team 

•  team  task  knowledge 

•  domain  expertise 

•  shared  understanding 

•  collaborative  knowledge 


•  collaborative  knowledge 

•  shared  understanding 


•  analyze,  revise  output 


•  goal  requirements 

•  exit  criteria 


Communication  Mechanism  for  Information  Processing  and  Knowledge  Building  (applies  to  all  stages): 

discussing  individual  information  •  discussing  team  generated  information 

negotiating  perspectives  •  providing  rationale  for  individual  solutions 


•  presenting  individual  information 

•  disagreement 


•  questioning 


discussion  of  possible  solutions 
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